Maximal relevance and optimal learning machines

Abstract We explore the hypothesis that learning machines extract representations of maximal relevance, where relevance is defined as entropy energy distribution internal representation. show mutual information between representation a machine and features it extracts from data bounded below by relevance. This motivates our study models with relevance—that we call optimal machines—as candidates maximally informative representations. analyse how maximisation constrained both architecture model used available data, in practical cases. find sub-extensive do not affect thermodynamics model, may significantly performance, criticality enhances but existence critical point necessary condition. On specific tasks, (i) values likelihood are achieved (ii) approach can be finite dataset (iii) associated broadening spectrum levels representation, agreement maximum hypothesis.